Method and apparatus for motion invariant imaging

ABSTRACT

Object motion during camera exposure often leads to noticeable blurring artifacts. Proper elimination of this blur is challenging because the blur kernel is unknown, varies over the image as a function of object velocity, and destroys high frequencies. In the case of motions along a 1D direction (e.g. horizontal), applicants show that these challenges can be addressed using a camera that moves during the exposure. Through the analysis of motion blur as space-time integration, applicants show that a parabolic integration (corresponding to constant sensor acceleration) leads to motion blur that is not only invariant to object velocity, but preserves image frequency content nearly optimally. That is, static objects are degraded relative to their image from a static camera, but all moving objects within a given range of motions reconstruct well. A single deconvolution kernel can be used to remove blur and create sharp images of scenes with objects moving at different speeds, without requiring any segmentation and without knowledge of the object speeds.

GOVERNMENT SUPPORT

The invention was supported, in whole or in part, by the followinggrants:

-   -   HM 1582-05-C-0011 from the National Geospatial Intelligence        Agency,    -   IIS-0413232 from the National Science Foundation, and    -   CAREER 0447561 from the National Science Foundation.

The government has certain rights in the invention.

BACKGROUND OF THE INVENTION

Motion blur often limits the quality of photographs and can be caused byeither the shaking of the camera or the movement of photographed objects(e.g., subject, passerby, props, etc.) in the scene. Modern camerasaddress the former case with image stabilization, where motion sensorscontrol mechanical actuators that shift the sensor or camera lenselement in real time during the exposure to compensate for the motion(shaking) of the camera, e.g. Canon. 2003. EF Lens Work III, The Eyes ofEOS. Canon Inc. Lens Product Group. The use of image stabilizationenables sharp hand-held photographs of still subjects at much longershutter speed, thereby reducing image noise. Unfortunately, imagestabilization only addresses camera motion and cannot help with movingobjects in the subject scene or field of view.

One option is to remove the blur after the shot was taken usingdeconvolution. However, this raises several challenges. First, thetypical motion-blur kernel is a line segment in the direction of motion,which corresponds to a box filter. This kernel severely attenuates highspatial frequencies and deconvolution quickly becomes ill-conditioned.Second, the length and direction of the blur kernel both depend on themotion and are therefore unknown and must be estimated. Finally, motionblur usually varies over the image since different objects or regionscan have different motion, and segmentation must be used to separateimage regions with different motion. These two later challenges leadmost existing motion deblurring strategies to rely on multiple inputimages (see Bascle, B., Blake, A., and Zisserman, A., “Motionde-blurring and superresolution from an image sequence,” ECCV, 1996;Rav-Acha and Peleg, S., “Two motion-blurred images are better than one,”Pattern Recognition Letters, 2005; Zheng, M. S. J., “A slit scanningdepth of route panorama from stationary blur,” Proc. IEEE Conf. Comput.Vision Pattern Recog., 2005; Bar, L., Berkels, B., Sapiro, G., andRumpf, M., “A variational framework for simultaneous motion estimationand restoration of motion-blurred video,” ICCV, 2007; Ben-Ezra, M., andNayar, S. K., “Motion-based motion deblurring,” PAMI, 2004; Yuan, L.,Sun, J., Quan, L., and Shum, H., “Image deblurring with blurred/noisyimage pairs,” SIGGRAPH, 2007.)

More recent methods attempt to remove blur from a single input imageusing natural image statistics (see Fergus, R., Singh, B., Hertzmann,A., Roweis, S., and Freeman, W., “Removing camera shake from a singlephotograph,” SIGGRAPH, 2006; Levin, A., “Blind motion deblurring usingimage statistics,” Advances in Neural Information Processing Systems(NIPS), 2006). While these techniques demonstrated impressive abilities,their performance is still far from perfect. Raskar et al. proposed ahardware approach that addresses the first challenge (Raskar, R.,Agrawal, A., and Tubmlin, J., “Coded exposure photography: Motiondeblurring using fluttered shutter,” ACM Transactions on Graphics,SIGGRAPH 2006 Conference Proceedings, Boston, Mass. vol. 25, pgs.795-804). A fluttered shutter modifies the line segment kernel toachieve a more broad-band frequency response, which allows fordramatically improved deconvolution results. While the Raskar approachblocks half of the light, the improved kernel is well worth thetradeoff. However, this approach still requires the precise knowledge ofmotion segmentation boundaries and object velocities, an unsolvedproblem.

SUMMARY OF THE INVENTION

The present invention addresses the foregoing problems in the art. Inthe present invention, applicants show that if the motion is restrictedto a 1D set of velocities, such as horizontal motion (as is the casewith many real world objects like cars or walking people), one canaddress all three challenges of the prior art mentioned above. Usingcamera hardware similar to that used for image stabilization, applicantsand the present invention make the point-spread function (PSF) invariantto motion and easy to invert. For this, applicants/the invention systemintroduce a specific camera movement during exposure. This movement isdesigned so that the compound motion of the camera and any objectvelocity (within a speed range and along the selected orientation) andat any depth in the camera field of view results in the sameeasy-to-invert PSF. Since the entire scene is blurred with an identicalPSF (up to tail truncation), including static objects and movingobjects, the blur can be removed via deconvolution, without segmentingmoving objects and without estimating their velocity. In practice,applicants find that motions even somewhat away from the selected 1Dorientation are deblurred as well.

In a preferred embodiment, a method and apparatus deblurs images of amoving object. During imaging of the moving object, the invention systemblurs an entire scene of the moving object. This blurring is in a mannerwhich is invariant to velocity of the moving object. Next the inventionsystem deconvolutes the blurred entire scene and generates areconstructed image. The reconstructed image displays the moving objectin a deblurred state.

The invention step of blurring the entire scene is preferablyimplemented by moving any one or combination of the camera, a lenselement of the camera and camera sensor. This moving of the camera orpart of the camera system includes linear movement in a range ofdirection and speed sufficient to include direction and speed of themoving object. Examples of linear movement here are a sinusoidalpattern, a parabolic pattern or any other simple harmonic motion and thelike.

In one embodiments, the linear movement follows a parabolic path bymoving laterally initially at a maximum speed of the range and slowingto a stop and then moving in an opposite direction laterally, increasingin speed to a maximum speed of the range in the opposite direction andstopping.

The present invention may be applied to a variety of moving objects andenvironments. For non-limiting examples, the moving object may be (a) amoving vehicle on a roadway or other terrain, (b) a body part of apatient (human or animal), (c) in aerial photography, or other movingobjects in a subject scene.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing will be apparent from the following more particulardescription of example embodiments of the invention, as illustrated inthe accompanying drawings in which like reference characters refer tothe same parts throughout the different views. The drawings are notnecessarily to scale, emphasis instead being placed upon illustratingembodiments of the present invention.

FIG. 1 a is a block diagram of an embodiment of the present invention.

FIG. 1 b is a flow diagram of an embodiment of the present invention.

FIGS. 2 a-2 d are schematic graphs illustrating xt-slice and integrationcurves resulting from different camera motions.

FIGS. 2 e-2 h are graphs illustrating corresponding integration curvessheared to account for object slope for FIGS. 2 a-2 d.

FIGS. 2 i-2 l are graphs of projected point spread functionscorresponding to different object velocities of FIGS. 2 a-2 d.

FIGS. 3 a-3 d is a set of simulated photographs of five dots moving overa range of speeds and directions.

FIGS. 4 a-4 d are illustrations of integration curve traces in spacetime and corresponding log spectrums of a static camera, a parabolicmotion camera of the present invention, a flutter shutter camera and theupper bound.

FIGS. 5 a-5 c are synthetic visualizations (imagery) of point spreadfunction information loss between a blurred input and deblurringsolution result.

FIG. 6 is a schematic view of one embodiment of the present invention.

FIGS. 7 a-7 c are photographic illustrations of deblurring results ofthe present invention.

FIGS. 8 a-8 e are comparisons of deblurring photographic images usingthe present invention and using static camera deblurring of the art.

FIGS. 9 a-9 b are scene views illustrating the present invention PSFclipping and a working velocities range.

DETAILED DESCRIPTION OF THE INVENTION

A description of example embodiments of the invention follows.

Applicants' approach is inspired by wavefront coding (Cathey, W., andDowski, R., “A new paradigm for imaging systems,” Applied Optics, No.41, pgs. 1859-1866, (1995)), where depth of field is improved bymodifying a lens to make the defocus blur invariant to depth and easy toinvert. While the cited work deals with wave optics and depth of field,applicants and the present invention consider geometric ray optics andremove 1D motion blur.

By analyzing motion blur as integration in a space time volume overcurves resulting from camera and object motion, applicants prove thatone integration curve that results in a motion-invariant PSF(point-spread function) is a parabola. This corresponds to constant 1Dacceleration of the camera, first going fast in one direction,progressively slowing down to a stop and then picking up speed in theother (opposite) direction. As a result for any object velocity within arange, there is always one moment during exposure where the camera isperfectly tracking (in speed and direction) the object. While the cameramotion is along a straight or lateral line, applicants call it“parabolic motion” because of the parabolic relationship betweenposition and time.

In addition to its invariance to object speed, the present invention PSFpreserves more high frequencies for moving objects than a normalexposure. This however comes at the cost of slightly degradedperformance for static objects. In fact, applicants show that even ifobject motions could be estimated perfectly, the type of PSFs resultingfrom the parabolic camera motion is near-optimal for a stabledeconvolution over a range of possible object speeds. This optimality isin the sense of minimizing the degradation of the reconstructed imagefor a range of potential object velocities. In a nutshell, applicantsshow that there is a fixed bandwidth budget for imaging objects atdifferent velocities. For example, a static camera spends most of thisbudget to achieve high-quality images of static objects, at the cost ofsevere blur for moving objects. Applicants' (the present invention's)design distributes this budget more uniformly, improving thereconstruction of all motion velocities, at the price of a slightlyworse reconstruction of the static parts.

There are three basic options for implementing this camera movement: atranslation of the full camera, a rotation of the camera, or atranslation of the sensor or lens element. For a commercial product, thelatter may be the best solution, and could be achieved with the existinghardware used for stabilization. However, if the focal length is not toowide-angle, a camera rotation is a good approximation to sensortranslation and is easier to implement as a prototype. Applicantsdemonstrate a prototype using camera rotation and show 1Dspeed-invariant deconvolution results for a range of 1D and even forsome 2D motions.

Motion Invariant Integration

In order to derive a motion-invariant photography scheme, the presentinvention characterizes motion blur as an integration in space-time.Applicants show that the effect of object motion can be characterized bya shear. Applicants also show that a camera parabolic movement isinvariant to shear in space-time and permits the removal of motion blur.

Space-time analysis of motion blur: The set of 2D images falling on adetector over time forms a 3D space-time volume of image intensities.Consider a 2D xt-slice through that 3D space-time volume. Each row inthis slice represents a horizontal 1D image, as captured by a staticpinhole camera with an infinitesimal exposure time.

For sufficiently small exposure, a first order approximation to theobject motion is sufficient, and the motion path is assumed to belinear. In this case, scene points trace straight lines in the xt-sliceand the slopes of these lines are a function of the object velocity anddepth. FIGS. 2 a-2 d demonstrate an xt-slice—the green (central area)object 15 is static, which means that it is invariant to time andresults in vertical lines in space time. The blue (right hand side) andred (left hand side) objects 17, 19 are moving in opposite directions,resulting in oblique lines. The slope of these lines correspond toimage-space object velocity.

Formally, the space-time function of an object moving at constantvelocity s is related to that of a static object by a shear, sincekinematics gives:x(t)=x(0)+st  (1)

Camera motion and integration: If the sensor is translating, the imagerecorded at time instance t is a shifted version of row t in thext-plane (the xt-plane represents the scene relative to a staticcamera). Thus, when the scene is captured by a translating sensor over afinite exposure time, the recorded intensities (the blurred image) arethe average of all shifted images seen during the exposure length, atall infitisimal time instances. That is, the sensor elements integratelight over curves in the xt-plane. The simplest case is a static camera,which integrate light over straight vertical lines 13 (FIG. 2 a). Asensor translating with constant velocity leads to slant straightintegration lines 21 (in FIG. 2 b the camera tracks the red object 19motion). The integration curve of a uniformly-translating sensor is asheared version of that of the static sensor, following the sameprinciple as the object-motion shear (but in the opposite direction).More complex sensor motion leads to more general curves. For exampleFIG. 2 c presents a parabola 23, obtained with translating sensorundergoing a parabolic displacement. If a shutter is fluttered duringexposure as in Raskar et al. 2006 (cited above), the integration curve25 is discontinuous as shown in FIG. 2 d.

Since the applicants only translate the sensor, the integration curvesthat applicants consider are spatially shift invariant. Applicantsdenote by L(x,t) the intensity of light rays in the xt-slice, I(x) theintensity of the captured image, f(t) the integration curve, and [−T,T]an integration interval of length 2T. The captured image can be modeledas:

$\begin{matrix}{{I(x)} = {\int_{T}^{T}{{L\left( {{{f(t)} + x},t} \right)}\ {\mathbb{d}t}}}} & (2)\end{matrix}$

The Point Spread Function: Denote by I⁰(x) an ideal instantaneouspinhole image I⁰(x)=L(x, 0). The movement of the camera creates motionblur. For a static object, applicants can model it as a convolution ofI⁰ with a Point Spread Function (PSF) φ₀:I=φ₀{circle around (×)}I⁰. Inthis case, φ₀ is simply the projection off along the time direction ontothe spatial line:

$\begin{matrix}{{\phi_{0}(x)} = {{\int_{t}^{\;}\delta_{f{(t)}}} =_{x}{\mathbb{d}t}}} & (3)\end{matrix}$where δ is a Dirac.

Now consider objects moving at speed s and seek to derive an equivalentPoint Spread Function φ_(s). We can reduce this to the static case byapplying a change of frame that “stabilizes” this motion, that is, thatmakes the space-time volume of this object vertical. We apply theinverse of the shear in Eq. 1, which is the shear in the oppositedirection, and the sheared curve can be expressed as:fs(t)=f(t)−st  (4)

Sheared curves are illustrated in FIGS. 2 e-2 h. As a result, the PSFφ_(s) for a moving object is the vertical projection of the shearedintegration curve f_(s). Equivalently, it is the oblique projection of falong the direction of motion.

FIGS. 2 i-2 l present the PSF 26, 27, 28, 29 of the three differentobjects 15, 17, 19, for each of the curves 13, 21, 23, 25, in FIGS. 2a-2 d. For example, if the integration curve is a straight line 13, 21the PSF 26, 27 is a delta function for objects whose slope matches theintegration slope, and a box filter for other slopes. The box width is afunction of the deviation between the object slope and integrationslope.

The analytic way to derive this projection is to note that the verticalprojection is the “amount of time” the curve f spent at the spatialpoint x—the slope of the inverse curve. That is, if g_(s)=f_(s) ⁻¹ isthe inverse curve, the PSF (the vertical projection) satisfies:φ_(s)(x)=g_(s)′(x).

Shear invariant curves: Applicants and the present invention derive acamera motion rule that leads to a velocity invariant PSF. One canachieve such effect if one devotes a portion of the exposure timetracing each possible velocity, and one spends an equal amount of timetracing each velocity, so that all velocities are covered equally. Thederivative of an integration curve representing such motion should belinear and therefore, a candidate curve is a parabola.

Applicants have shown that PSFs corresponding to different velocitiesare obtained from sheared version of the sensor integration curve.Consider a parabola curve 23 of the form: f(t)=a₀t² (FIG. 2 c). Theresulting PSF 28 behaves like 1/√{square root over (a₀x)} (FIG. 2 k).Applicants note that a sheared parabola is also a parabola with the samescale, only the center is shifting

$\begin{matrix}{{f_{x}(t)} = {{{f(t)} - {st}} = {{a_{0}\left( {t - \frac{s}{2a_{0}}} \right)}^{2} - \frac{s^{2}}{4a_{0}}}}} & (5)\end{matrix}$

Thus, the projections φ_(s) are also identical up to a spatial shift.The important practical application of this property is that if thecamera is moved during integration along a parabola curve, one candeconvolve all captured images I with the same PSF, without segmentingthe moving objects in the image, and without estimating their velocityor depth. The small spatial shift of the PSF leads to a small spatialshift of the deconvolved image, but such a shift is uncritical as itdoes not translate to visual artifacts. This simply means that theposition of moving objects corresponds to different time instants withinthe exposure. The time shift for a given velocity corresponds to thetime where the sensor is perfectly tracking this velocity.

It is noted that the above invariance involves two approximations. Thefirst approximation has to do with the fact that the invariantconvolution model is wrong at the motion layer boundaries. However, thishas not been a major practical issue in applicants' experiments and isvisible only when both foreground and background have high-contrasttextures. The second approximation results from the fact that a parabolais perfectly shear invariant only if an infinite integration time isused. For any finite time interval, the accurate projection is equal to:

$\begin{matrix}{{\phi_{s}(x)} = \left\{ \begin{matrix}\frac{1}{\sqrt{a_{0}\left( {x + \frac{s^{2}}{4a_{0}}} \right)}} & {{{for}\mspace{14mu} 0} \leq {x + \frac{s^{2}}{4a_{0}}} \leq {a_{0}\left( {T - \frac{s}{2a_{0}}} \right)}^{2}} \\\frac{1}{2\sqrt{a_{0}\left( {x + \frac{s^{2}}{4a_{0}}} \right)}} & {{{for}\mspace{14mu}{a_{0}\left( {T - \frac{s}{2a_{0}}} \right)}^{2}} \leq {x + \frac{s^{2}}{4a_{0}}} \leq {a_{0}\left( {T + \frac{s}{2a_{0}}} \right)}^{2}} \\0 & {otherwise}\end{matrix} \right.} & (6)\end{matrix}$

Thus, for a finite integration interval, the tails of the PSF do dependon the slope s. This change in the tail clipping can also be observed inthe projected PSFs 28 in FIG. 2 k. For a sufficiently bounded range ofslopes the tail clipping happens far enough from the center and itseffect could be neglected. However, Equation 6 also highlights thetradeoffs in the exact parabola scaling a₀. Smaller a₀ values lead to asharper PSF. On the other hand, for a given integration interval [−T,T], the tail clipping starts at

${\sqrt{a}\left( {T - \frac{s}{2a_{0}}} \right)};$thus reducing a₀ also reduces the range of s values for which the tailclipping is actually negligible.

Simulation: To simulate the blur from a camera moving in a parabolicdisplacement in space-time (constant 1D acceleration), applicantsprojected synthetic scenes and summed displaced images over the cameraintegration time. FIG. 3 a shows five dots at the initial time, withtheir motion vectors, and FIG. 3 b shows their final configuration atthe end of the camera integration period. FIG. 3 c is the photographobtained with a static camera, revealing the different impulse responsesfor each of the five different dot speeds. FIG. 3 d shows the image thatwould be recorded from the camera undergoing a parabolic displacement.Note that now each of the dots creates virtually the same impulseresponse, regardless of its speed of translation (there is a smallspeed-dependent spatial offset to each kernel). This allows an unblurredimage to be recovered from spatially invariant deconvolution.

Optimality

Applicants derive optimality criteria as follows.

Upper Bound

We have seen that the parabola is a shear invariant curve, and thatparabolic displacement is one camera movement that yields a PSFinvariant to motion. Here applicants show that, in the case of 1Dmotions, this curve approaches optimality even if we drop themotion-invariant requirement. That is, suppose that we could perfectlysegment the image into different motions, and accurately know the PSF ofeach segment. For good image restoration, we want the PSFs correspondingto different velocities or slopes to be as easy to invert as possible.In the Fourier domain, this means that low Fourier coefficients must beavoided. We show that, for a given range of velocities, the ability tomaximize the Fourier spectrum is bounded and that our parabolicintegration approaches the optimal spectrum bound.

At a high level, our proof is a bandwidth budget argument. We show that,for a given spatial frequency w_(x), we have a fixed budget which mustbe shared by all motion slopes. A static camera spends most of thisbudget on static objects and therefore does poorly for other objectspeeds. In contrast, our approach attempts to distribute this budgetuniformly across the range of velocities and makes sure that nocoefficient is low.

Space time integration in the frequency domain: We consider the Fourierdomain ω_(x), ω_(t) of a scanline of space time. Fourier transforms willbe denoted with a hat and Fourier pairs will be denoted k

{circumflex over (k)}.

First consider the space-time function of a static object. It isconstant over time, which means that its Fourier transform is non-zeroonly on the pure spatial frequency line ω_(t)=0. This line is the 1DFourier transform of the ideal instantaneous image I⁰.

We have seen that image-space object velocity corresponds to the slopeof a shear in space time. In the frequency domain, a given slopecorresponds to a line orthogonal to the primal slope. Or equivalently,the shear in the primal corresponds to a shear in the opposite directionin the Fourier domain. The frequency content of an object at velocity sis on the line of slope s going through the origin. A range ofvelocities −S≦s≦S corresponds to a double-wedged in the Fourier domain.This is similar to the link between depth and light field spectra (Chai,J., Tong, X., Chan, S., and Shum, H., “Plenoptic sampling,” SIGGRAPH,2000; Isaksen, A., McMillan, L., and Gortler, S. J., “Dynamicallyreparameterized light fields,” SIGGRAPH, 2000). This double-wedged isthe frequency content that we strive to record. Areas of the Fourierdomain outside it correspond to faster motion, and can be sacrificed.

Consider a given light integration curve f and its 2D trace k(x,t) inspace time, where k(x,t) is non zero only at x=f(t) (FIGS. 4 a-4 d). The1D image scanline can be seen as the combination of a 2D convolution inspace time by k, and a 2D (two-dimensional) to 1D (one-dimensional)slicing. That is, we lookup the result of the convolution only at timet=0. The convolution step is key to analyzing the loss of frequencycontent. In the Fourier domain, the convolution by k is a multiplicationby its Fourier transform {circumflex over (k)}.

For example, a static camera has a kernel k that is a box in time, timesa Dirac in space. Its Fourier transform is a sinc in time, times aconstant in space (FIG. 4 a). The convolution by k results in areduction of the high temporal frequencies according to the sinc. Sincefaster motion corresponds to larger slopes, their frequency content ismore severely affected, while a static object is perfectly imaged.

In summary, we have reduced the problem to designing an integrationcurve whose spectrum {circumflex over (k)} has the highest possibleFourier coefficients in the double-wedge defined by a desired velocityrange.

Slicing: We now show that for each vertical slice of the Fourier doubledwedge, we have a fixed bandwidth budget because of conservation ofenergy in the spatial domain. That is, the sum of the squared Fouriercoefficients for a given spatial frequency ω_(x) is bounded from above.

When studying slices in the Fourier domain, we can use the slicingtheorem. First consider the vertical Fourier slice {circumflex over(k)}₀ going through (0,0). In the primal space time, this Fourier slicecorresponds to the projection along the horizontal x direction.{circumflex over (k)} ₀(ω_(t))

k _(p)(t)=∫_(x) k(x,t)dxAnd using the shifting property, we obtain an arbitrary slice for agiven ω_(x) using{circumflex over (k)}ω _(x)(ω_(t))

∫_(x) k(x,t)e ^(−2πiω) ^(x) ^(x) dxwhich only introduces phases shifts in the integral.

Conservation of energy: We have related slices in the Fourier domain tospace-only integrals of our camera's light integration curve inspace-time. In particular, the central slice is the Fourier transform ofk_(p)(t), the total amount of light recorded by the sensor at a givenmoment during the exposure. Conservation of energy imposesk _(p)(t)≦1  (8)

Since k is non-zero only during the 2T exposure time, we get a bound onthe square integral∫_(t) k _(p)(t)² dt≦2T  (9)

This bound is not affected by the phase shift used to extract slices atdifferent ω_(x).

Furthermore, by Parseval's theorem, the square integral is the same inthe dual and the primal domains. This means that for each slice at aspatial frequency ω_(x),

$\begin{matrix}{{\int_{\omega_{t}}{{{\hat{k}}_{\omega_{x}}\left( \omega_{t} \right)}^{2}{\mathbb{d}\omega_{t}}}} = {\int_{t}{\left\lbrack {{k_{p}(t)}{\mathbb{e}}^{{- 2}{\pi\omega}_{x}x}} \right\rbrack^{2}{\mathbb{d}t}}}} & (10) \\{\leq {2T}} & (11)\end{matrix}$

The squared integral for a slice is bounded by a fixed budget of 2T. Inorder to maximize the minimal frequency response, one should use aconstant magnitude. Given the wedged shape of our velocity range in theFourier domain, we get

$\begin{matrix}{{\min\limits_{\omega_{t}}{{{\hat{k}}_{\omega_{x}}\left( \omega_{t} \right)}}^{2}} \leq \frac{T}{S{\omega_{x}}}} & (12)\end{matrix}$where S is the absolute maximal slope (speed). This upper bound isvisualized in FIG. 4 d. In other words, if we wish to maximize thespectrum of the PSFs over a finite slope range −S≦s≦S, Eq 12 provides anupper bound on how much we can hope to achieve.

When one wants to cover a broader range of velocities, the budget mustbe split between a larger area of the Fourier domain and overallsignal-noise ratio is reduced according to a square root law.

Discussion of Different Cameras

Applicants have shown that in a traditional static camera, the lightintegration curve in space-time k(x,t) is a vertical box function.Performances are perfect for the ω_(t)=0 line corresponding to staticobjects, but degrade according to a sinc for lines of increasing slope,corresponding to higher velocities.

The flutter-shutter approach adds a broad-band amplitude pattern to astatic camera. The integration kernel k is a vertical 1D function over[−T,T] and the amount of recorded light is halved. Because of the lossof light, the vertical budget is reduced from 2T to T for each ω_(x).Furthermore, since k is vertical, its Fourier transform is constantalong ω_(x). This means that the optimal flutter code must have constantspectrum magnitude over the full domain of interest (FIG. 4 c). This iswhy the spatial resolution of the camera must be taken into account. Theintersection of the spatial bandwidth Ω_(max) of the camera and therange of velocities defines a finite double-wedge in the Fourier domain.The minimum magnitude of the slice at Ω_(x) is bounded by

$\frac{T}{2S\;\Omega_{x}}.$Since {circumflex over (k)} is constant along ω_(x), this bound appliesto all ω_(x). As a result, for all band frequencies |ω_(x)|<Ω_(max),{circumflex over (k)} spends energy outside the slope wedge and thusdoes not make a full usage of the vertical {circumflex over (k)}ω_(x),budget.

The parabolic integration curve attempts to distribute the bandwidthbudget equally for each ω_(x) slice (FIG. 4 b), resulting in almost thesame performance for each motion slope in the range, but a falloffproportional to 1/√{square root over (ω_(x))} along the spatialdimension. To see why, applicants note that using some calculusmanipulation the Fourier transform of an infinite parabola can becomputed explicitly. If the integration kernel k is defined via the(infinite) parabola curve f(t)=a₀t², then

$\begin{matrix}{{\hat{k}\left( {\omega_{x},\omega_{t}} \right)} = {\begin{matrix}1 \\\sqrt{2a_{0}\omega_{x}}\end{matrix}{\mathbb{e}}^{{\mathbb{i}}\frac{\omega_{t}^{2}}{4a_{0}\omega_{x}}}}} & (13)\end{matrix}$

On the other hand, achieving a good PSF for all −S≦s≦S implies that

$a_{0} \geq \frac{S}{2T}$(otherwise, from Eq 6 the PSF won't include an infinite spike). Using Eq13, applicants can conclude that if the exposure was infinitely long

${{\hat{k}\left( {\omega_{x},\omega_{t}} \right)}}^{2} = \frac{T}{S{\omega_{x}}}$and the infinite parabola has the same falloff as the upper bound. Ofcourse, the upper bound is valid for a finite exposure, and we canrelate the infinite parabola to our finite kernel by a multiplication bya box in the spatial domain, which is a convolution by a sinc in theFourier domain. Thus, the parabola curve of finite exposures approachesthe upper bound, in the limit of long exposure times. The intuitivereason why the parabolic camera can better adapt to the wedged shape ofthe Fourier region of interest is that its kernel is not purelyvertical, that is, the sensor is moving. A parabola in 2D contains edgepieces of different slopes, which corresponds to Fourier components oforthogonal orientation.

In summary, in the special case of 1D motions, if one seeks the abilityto reconstruct a given range of image-space velocities, with minimaldegradation to the reconstructed image for any velocity, the paraboliclight integration curve is near optimal. On the other hand, a flutteredshutter can handle all motion directions, albeit at the cost of motionidentification and image segmentation.

Simulation: To visualize these tradeoffs, in FIGS. 5 a-5 c, applicantssynthetically rendered a moving car. Applicants simulate a static camera(FIG. 5 a), a parabolic displacement (FIG. 5 b), and a static camerawith a flutter-shutter (FIG. 5 c), all with an identical exposure lengthand an equal noise level. The box-deblurred car in FIG. 5 a lost highfrequencies. The flutter-shutter reconstruction (FIG. 5 c) is muchbetter, but the best results are obtained by the parabolic blur (FIG. 5b) of the present invention.

A static camera, the flutter shutter camera, and a parabolic motioncamera (the present invention) each offer different performancetradeoffs. A static camera is optimal for photographing static objects,but suffers significantly in its ability to reconstruct spatial detailsof moving objects. A flutter shutter camera is also excellent forphotographing static objects (although records a factor of two lesslight than a full exposure). It provides good spatial frequencybandwidth for recording moving objects and can handle 2D motion.However, to reconstruct, one needs to identify the image velocities andsegment regions of uniform motion. Motion-invariant photography of thepresent invention (e.g., parabolic motion camera) requires no speedestimation or object segmentation and provides nearly optimalreconstruction for the worst-case speed within a given range. However,relative to the static camera and the flutter shutter camera, it givesdegraded reconstruction of static objects. While the invention method isprimarily designed for 1D motions, we found it gave reasonablereconstructions of some 2D motions as well.

Embodiments of the present invention 11 are thus as illustrated in FIGS.1 a and 1 b. With reference to FIG. 1 a, a camera system 103 includes anautomated control assembly 101 (such as that in FIG. 7 or similar) and acamera 100. The camera 100 has (i) a lens and/or lens elements forfocusing and defining the field of view 125 and (ii) an optional motionsensor. The automated control assembly 101 has a motor and/orelectro-mechanical mechanism for moving the camera 100, lens, lenselement(s) or sensor as prescribed by the present invention.

In particular, the automated control assembly 101 operates the camera100 to produce a subject image having both moving objects 10 and staticobjects 20. First as shown at step 201 in FIG. 1 b, the control assembly101 operates the camera shutter together with moving the camera 100 (asa whole, or just the lens or sensor) preferably in a lateral parabolicmotion. Other linear movement such as in a sinusoidal pattern or simpleharmonic motion or the like are suitable. The result is an entire sceneblurred 110 with a single point spread function (PSF) throughout. Asobtained at step 203, the blurred entire scene (i.e., working orintermediate image) 110 is invariant to object motion of the movingobjects 10. Thus, a deconvolution processor 105 reconstructs the subjectimage with one PSF and without requiring a velocity or depth of themoving object 10. Step 205 is illustrative.

The deconvolution processor 105 may be within the camera 100 processingduring image exposure (near real time) or external to the camera 100processing subsequence to image exposure. Deconvolution processor (orsimilar engine) 105 employs deconvolution algorithms and techniquesknown in the art. Example deconvolution techniques are as given inLevin, A., Fergus, R., Durand, F., and Freeman, W., “Image and depthfrom a conventional camera with a coded aperture,” SIGGRAPH, 2007; andLucy, L., “Bayesian-based iterative method of image restoration,”Journal of Ast., 1974, both herein incorporated by reference.

The reconstructed image 112 showing moving objects 10 and static objects20 in an unblurred state is produced (generated as output).

Experiments

While camera stabilization hardware should be capable of moving adetector with the desired constant acceleration (parabolic displacement)inside a hand-held camera, applicants chose to use larger scalestructures for an initial prototype, and approximate sensor translationusing a rotation of the entire camera. The hardware shown in FIG. 6rotates the camera 61 in a controlled fashion to evaluate the potentialof motion-invariant photography. The camera (for instance a Canon EOS 1DMark II with an 85 mm lens) sits on a platform 63 that rotates about avertical axis through the camera's optical center. Applicants use a camapproach to precisely control the rotation angle over time. A rotatingcam 67 moves a lever 65 that is rigidly attached to the camera 61 togenerate the desired acceleration. For θ∈[−π,π] the cam 67 edge isdesigned such that the polar coordinate radius is a parabola:x(θ)=cos(θ)(c−bθ ²)y(θ)=sin(θ)(c−bθ ²)  (14)

In applicants' experiments, c=8 cm, b=0.33 cm. The cam 67 rotates at aconstant velocity, pushing the lever arm 65 to rotate the camera 61 withapproximately constant angular acceleration, yielding horizontal motionwith the desired parabolic integration path in space-time. For a fixedcam size, one can increase the magnitude of the parabola by moving thecam 67 closer to the camera 61. Applicants place a static camera next tothe rotating camera 61 to obtain a reference image for eachmoving-camera image. A microcontroller synchronizes the cam 67 rotationand the camera shutters. In order to reduce mechanical noise, theexposure length of the system was set to 1 second. This relatively longexposure time limits the linear motion approximation for some real-worldmotions. To calibrate the exact PSF produced by the rotating camera 61,applicants captured a blurred image I of a calibration pattern.Applicants also captured a static image I₀ of the same pattern andsolved for the PSF φ minimizing the squared convolution error: φ=argmin∥I⁰−φ*I∥².

Results

The deconvolution results presented herein were achieved with the sparsedeconvolution algorithm of Levin, A., Fergus, R., Durand, F., andFreeman, W., “Image and depth from a conventional camera with a codedaperture,” SIGGRAPH, 2007 (incorporated herein by reference).Comparable, but slightly worse, results can be obtained using theRichardson-Lucy deconvolution algorithm (see Lucy, L., 1974, citedabove).

FIGS. 7 a-c present deblurring results on images captured by theinvention camera 100, 61. For each example, FIGS. 7 a, 7 b, 7 c presentthe scene (top row) captured by a synchronized static camera and thedeconvolution results (bottom row) obtained from the invention movingcamera 100, 61. In the first pair of images (FIG. 7 a), the movingobjects were mounted on a linear rail to create multiple velocities fromthe multiple depth layers. The other pairs of images (FIGS. 7 b, 7 c)involved natural human motions. The invention approach (bottom row FIGS.7 a-7 c) deblurs the images reasonably well despite the fact that thehuman motions were neither perfectly linear nor horizontal. The middlepair of images (FIG. 7 b) shows multiple independent motions in oppositedirections, all deblurred well in the present invention (bottom FIG. 7b). The far-right image pair (FIG. 7 c) had many non-horizontal motions,resulting from the man (moving subject) walking toward the camera. Whilethe face contains some residual blur, the deconvolved image has fewobjectionable artifacts.

Applicants are encouraged by the deconvolution results on some imageseven with substantial non-horizontal motions. One possible explanationfor the results is the aperture effect, the ambiguity of the 2D motionof locally 1-dimensional image structures, such as edges and contours.The velocity component normal to the edge or contour is determined fromthe image data, but the parallel component is ambiguous. Local imagemotion that could be explained by horizontal motions within the range ofthe camera motions should deconvolve correctly, even though the objectmotions were not horizontal.

Note that the static object resolution in applicants results decreaseswith respect to the static camera input, as applicant uniformlydistribute the bandwidth budget over velocities range.

Motion deblurring from a stationary camera is very challenging due tothe need to segment the image by motion and estimate accurate PSFswithin each segment. FIGS. 8 a-8 e demonstrate these challenges. Forexample, we can try to deconvolve the image with a range of box widths,and manually pick the one producing the most visually plausible results(FIG. 8 b). In the first row of images (FIG. 8 a), static camera imagesare presented. Here deconvolving with the correct blur can sharpen themoving layer, but creates significant artifacts in the static parts, andan additional segmentation stage is required. In the second case, imagesof FIG. 8 b show a box filter filtered manually to the moving layer andapplied to deblur the entire image. As a result, most blurred edges areocclusion boundaries and even manually identifying the correct blurkernel is challenging. In FIG. 8 c-8 d, results of a recent automaticalgorithm (Levin 2006, cited above) are shown. The images of FIG. 8 cemployed layers segmentation by Levin 2006, and the images of FIG. 8 dshow deblurring results of Levin 2006. While this algorithm did areasonable job for the left-hand side image (which was captured using alinear rail) it did a much worse job on the human motion in theright-hand side image.

The present invention results obtained spatially uniform deconvolutionof images (FIG. 8 e) from parabolic integration.

In FIG. 9, applicants illustrate the effect of the PSF tail clipping(discussed above in FIG. 2 k and Eq 6) on the valid velocities range.Applicants used the invention parabolic camera to capture a binarypattern. The pattern was static in the first shot (top row FIGS. 9 a-9b) and for the other two (middle and bottom rows FIGS. 9 a-9 b) linearlymoving during the exposure. As shown in the top image of FIG. 9 b, thestatic motion was easily deblurred, and the PSF approximation isreasonable for the slow motion case as well. For the faster motion case(middle and bottom images FIG. 9 b), deconvolution artifacts start to beobserved, as effect of the clipping of the tails of the PSF becomesimportant and the static motion PSF is not an accurate approximation forthe moving object smear.

Accordingly, the present invention suggests a solution that handlesmotion blur along a 1D direction. In the invention system 11, the camera100 (FIGS. 1 a, 1 b) translates within its exposure following aparabolic displacement rule. The blur resulting from this special cameramotion is shown to be invariant to object (moving and/or static) depthand velocity. Hence, blur can be removed by deconvolving the entireimage with an identical, known PSF. This solution eliminates the majortraditional challenges involved with motion debluring: the need tosegment motion layers and estimate a precise PSF in each of them.Furthermore, the present invention analyzes the amount of informationthat can be maintained by different camera paths, and show that theparabola path approaches the optimal PSF whose inversion is stable atall velocities.

The teachings of all patents, published applications and referencescited herein are incorporated by reference in their entirety.

While this invention has been particularly shown and described withreferences to example embodiments thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade therein without departing from the scope of the inventionencompassed by the appended claims.

For example, the above description mentions parabolic or other motion ofthe camera. This motion may be manually produced by the user(photographer) in some embodiments and mechanically automated, such asby a controller assembly 101, in other embodiments.

Further, the controller 101 and deconvolution processor 105 may beexecuted by one CPU (digital processing unit) or chip in camera 100 ormay be separate computers/digital processing systems, either stand aloneor networked. Common computer network communications, configurations,protocols, busses, interfaces, couplings (wireless, etc.) and the likeare used. The network may be a wide area network, a local area network,a global computer network (e.g., Internet) and the like.

Fields of application may then include:

(a) medical imaging at hospitals, clinics, care profession's office(e.g., dentist, pediatrician, etc.), mobile unit, etc;

(b) aerial imaging of vehicles or vessels in various terrain maneuvers;

(c) monitoring systems at airports, secured areas, security locations,or public/pedestrian throughways; and

(d) toll booth imagery of traveling vehicle license plates or similarcheck points where a moving subject is photographed for identityverification or similar purposes.

Other applications and uses are within the purview of one skilled in theart given this disclosure.

What is claimed is:
 1. A method for deblurring images of a movingobject, comprising: during imaging of a moving object, the moving objectcausing blur in an image, blurring an entire scene that includes themoving object, said blurring being invariant to the velocity of themoving object; deconvolving the blurred entire scene with a point spreadfunction that is invariant to the velocity of the moving object togenerate a reconstructed image, the reconstructed image bearing themoving object in a deblurred state, wherein the step of blurring theentire scene is in a manner enabling integration in a space and timevolume that results in the point spread function being invariant tovelocity of the moving object; wherein the step of blurring blurs theentire scene, including static objects and the moving object, andwherein the step of deconvolving removes blur from the blurred entirescene using the point spread function, without segmenting the movingobject and without estimating velocity of the moving object.
 2. A methodas claimed in claim 1 wherein the step of blurring the entire scene isimplemented by moving any one or combination of a camera, a lens elementof the camera, and a sensor of the camera.
 3. A method as claimed inclaim 2 wherein the step of moving includes movement in a range ofdirection and speed sufficient to include direction and speed of themoving object.
 4. A method as claimed in claim 3 wherein movement is anyof: a sinusoidal pattern, a parabolic pattern, a simple harmonic motion,and a sweep motion starting in one direction and progressivelydecelerating then accelerating in an opposite direction.
 5. A method asclaimed in claim 4 wherein the movement follows a parabolic path bymoving laterally initially at a maximum speed of the range and slowingto a stop and then moving in an opposite direction laterally, increasingin speed to a maximum speed of the range in the opposite direction andstopping.
 6. A method as claimed in claim 1 wherein the moving object isa moving vehicle on a roadway.
 7. A method as claimed in claim 1 whereinthe moving object is a part of a mammalian body.
 8. A method as claimedin claim 1 wherein the deconvolution is computed near real-time of imageexposure.
 9. A method as claimed in claim 1 wherein the deconvolution isby an application separate from and subsequent to image exposure by acamera.
 10. A method as claimed in claim 1 wherein the reconstructedimage bears multiple moving objects, each in a deblurred state.
 11. Anapparatus for deblurring images of a moving object, comprising: a cameracontroller, during imaging of a moving object, the moving object causingblur in images, the controller effectively moving a camera to blur anentire scene that includes the moving object and said blurring of theentire scene being in a manner that is invariant to the velocity of themoving object; and a deconvolution processor coupled to receive theblurred entire scene and deconvolving the blurred entire scene with apoint spread function that is invariant to the velocity of the movingobject, resulting in a reconstructed image bearing the moving object ina deblurred state, wherein the camera controlled blurring the entirescene is in a manner enabling integration in a space and time volumethat results in the point spread function being invariant to velocity ofthe moving object; wherein the step of blurring blurs the entire scene,including static objects and the moving object, and wherein the step ofdeconvolving removes blur from the blurred entire scene using the pointspread function, without segmenting the moving object and withoutestimating velocity of the moving object.
 12. Apparatus as claimed inclaim 11 where the controller effectively moving the camera moves anyone or combination of the camera, a lens element of the camera and asensor of the camera.
 13. Apparatus as claimed in claim 11 wherein thecontroller effectively moving the camera includes movement in a range ofdirection and speed sufficient to include direction and speed of themoving object.
 14. Apparatus as claimed in claim 13 wherein movement isany of: a sinusoidal pattern, a parabolic pattern, a simple harmonicmotion, and a sweep motion starting in one direction and progressivelydecelerating then accelerating in an opposite direction.
 15. Apparatusas claimed in claim 14 wherein the movement follows a parabolic path bymoving laterally initially at a maximum speed of the range and slowingto a stop and then moving in an opposite direction laterally, increasingin speed to a maximum speed of the range in the opposite direction andstopping.
 16. Apparatus as claimed in claim 11 wherein the moving objectis any of a moving vehicle on a roadway, a body part of a patient, and aface of a subject.
 17. Apparatus as claimed in claim 11 wherein thedeconvolution processor deconvolutes the blurred entire scene nearreal-time of image exposure.
 18. Apparatus as claimed in claim 11wherein the deconvolution processor executes as an application separatefrom and subsequent to image exposure by the camera.
 19. Apparatus asclaimed in claim 11 wherein the reconstructed image bears multiplemoving objects, each in a deblurred state.
 20. A motion invariantimaging system, the system imaging moving comprising: controller meansfor effectively (i) controlling a camera during imaging of a movingobject, the moving object causing blur in an image, and (ii) blurring anentire scene that includes the moving object; and deconvolution meansfor deconvolving the blurred entire scene with a point spread functionthat is invariant to the velocity of the moving object to generate areconstructed image bearing the moving object in a deblurred state;wherein blurring the entire scene includes blurring static objects andthe moving object using the point spread function such that thedeconvolving removes blur from the blurred entire scene using the pointspread function, without segmenting the moving object and withoutestimating velocity of the moving object, and wherein the controllermeans blurring the entire scene is in a manner enabling integration in aspace and time volume that results in the point spread function beinginvariant to velocity of the moving object; wherein the step of blurringblurs the entire scene, including static objects and the moving object,and wherein the step of deconvolving removes blur from the blurredentire scene using the point spread function, without segmenting themoving object and without estimating velocity of the moving object.